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RE: Final Decision in Response to Amplify’s Protest 

Dear Mr. Armbruster: 

Istation is the best diagnostic tool for the state of North Carolina. Istation received the 
contract award through a fair and objective process conducted by an evaluation committee that 
adhered to all laws, rules, and policies. The evaluation committee recommended Istation to me and 
to the State Board of Education (the “State Board”). I approved that recommendation, and the State 
Board unanimously approved that recommendation. 

Pursuant to 09 NCAC 06B .1102(c)(2), this letter and its attachments constitute my decision 
on behalf of the North Carolina Department of Public Instruction (the “Department”) to the protest 
by your client, Amplify Education, Inc. (“Amplify”), of the Department’s June 7, 2019 Read to 
Achieve contract award to Imagination Station, Inc. (for this letter, “Istation” refers to the company 
or Imagination Station Inc.’s reading diagnostic tool, Istation, as the context requires). 

I have reviewed the claims in your protest letter, the claims expressed during the in-person 
protest meeting, and the claims in your supplemental letter. After careful review, I stand by the 
Department’s original contract award and deny Amplify’s protest. 

This letter will give you and the public the clarity necessary to understand the entirety of this 
procurement process. Amplify’s claims about Imagination Station, Inc.’s reading diagnostic tool, 
Istation, and its claims about the procurement process, are incorrect and many are based on 
misstatements of fact. 

Please consider these four pages a summary. Details regarding each section can be found in 
the provided attachments. 

I. Read to Achieve Diagnostic Tool Procurement 

North Carolina law requires one reading diagnostic tool for every kindergarten, first, second, 
and third grade class. This same law directed the Department to conduct a Request for Proposal 
(“RFP”), consider certain factors, and select one tool to be implemented beginning with the 2019- 
2020 school year. I was responsible for assembling an evaluation committee of Department 
employees who would recommend a tool for my approval. (Please see Attachment A). 
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The State Board, the Department, and I awarded this contract to Istation because it is the 
best diagnostic tool for North Carolina. 

• Istation provides immediate results for parents to view, track, and better help their 
students’ literacy progress; even allowing families to log in at home. 

• Istation provides immediate data and results for teachers and administrators to view, 
analyze, and personalize their teaching to better meet the needs of North Carolina 
students. 

• Istation minimizes the amount of instructional time impacted by assessments. Put another 
way, teachers can spend more time teaching, rather than administering tests. 

• Istation is a highly integrated tool that provides assessment results with instructional 
support for teachers and students. 

• Istation provides the Department and schools across North Carolina with the best value. 

• Istation provides more options for less money than does your client. 

This list goes on. 

Unfortunately, this positive story for North Carolina teachers and students has been 
overlooked due to misstatements of fact from your client, unintentional or not, and inappropriate 
actions by employees no longer with the Department. 

At the start of the process in 2017, evaluation committee members promised to prioritize 
fairness and objectivity over improper biases; going so far as to sign non-disclosure agreements 
and statements affirming that they had disclosed all possible conflicts of interest. Unfortunately, 
a small number of career Department employees did not honor these commitments. 

• One did not disclose that she had previously been paid by Amplify, her most recent 
employer before the Department, even though she signed the no conflict of interest 
document. 

• One violated the non-disclosure agreement to leak updates to outsiders about confidential 
issues. 

• A whistleblower provided evidence of a text message discussion detailing how 
committee members had voted and characterizing my priorities and efforts for educators 
as attempts to “appease lazy ass teachers.” (Their words, not mine.) 

• Members of the selection committee employed biased procedures that benefited the 
incumbent vendor (Amplify) over other vendors by not clarifying questions on 
specifications. If the questions had been clarified, as required by the procurement rules 
and procedures, it would have benefitted other vendors. 

• Members of the selection committee employed biased procedures that benefited the 
incumbent vendor (Amplify) over other vendors by allowing untrue statements or non- 
factual opinions to be presented as fact during their ranking process. If factually correct 
statements had been presented, it would have benefitted other vendors. 

• Members of the selection committee violated procurement policies outlined in the 
Department of Information Technology procurement manual by improperly adding 
weight to select criteria when ranking the vendors. 
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These employees are no longer with the Department. And, to be clear, the majority of 
career Department employees are honorable civil servants who place the needs of educators and 
students as their top priority. Please do not let the bad actions of a small minority portray all 
Department employees in a negative light. 

Regrettably, so as to conduct a fair and unbiased procurement process, the Department 
had to cancel two RFPs. (Please see Attachment B). Pursuant to state law and procurement rules 
and polices, and in consultation with the Department of Information Technology and the State 
Board, the Department entered negotiations with Amplify and Istation. (Please see Attachment 
C). 

The evaluation committee for the negotiations process presented to me its unanimous 
recommendation to select Istation, and I approved the recommendation to select Istation. 
Department staff then presented the recommendation to the State Board, and the State Board 
unanimously approved the recommendation to select Istation. For clarity, there was only one 
recommendation in the entire procurement process, and it was Istation. (Please see Attachment 
B). 

II. Amplify’s Incorrect Claims about Istation 

Throughout this protest for Amplify, you have made incorrect claims about Istation to 
argue in favor of Amplify’s request. Istation has already been used successfully in other states 
and more than satisfies the requirements to be the reading diagnostic tool for North Carolina. 
(Please see Attachment D). Contrary to your incorrect claims, and without limitation: 

• Istation satisfied the requirements of the procurement. 

• Istation meets the requirements of law. 

• Istation predicts students who are at risk of a reading difficulty. 

• Istation screens for dyslexia. 

• Istation is developmental^ appropriate. 

• Istation assesses required measures. 

• Istation is effective at measuring a student’s progress. 

• Istation allows for students to interact with and read aloud to teachers. 

• Istation provides sufficient data to allow detenninations of short-tenn student progress. 

• Istation accurately and reliably detennines which students will be reading on grade level 
by the end of the school year. 

• Istation is already on track for successful implementation in North Carolina. 

• Istation provides the Department and schools across North Carolina with the best value. 

III. Issues with Amplify and mCLASS in North Carolina 

In your protest letter, you state that the Department “must also consider factors including 
whether the vendor complies with industry standards, the vendor’s past perfonnance with the 
state, and the probability of the vendor providing the required service on time.” Amplify has 


3 




presented problems meeting the needs and requirements for the Department and North Carolina 
school systems. (Please see Attachment E). 

IV. Final Decision 

As described and detailed in the attachments, the claims in your protest letter that the 
award to Istation does not meet the requirements of the procurement process or North Carolina 
law are incorrect. 

The Department and the State Board followed all laws, policies, and rules related to this 
procurement process and contract award. Because of the reasons I have incorporated herein, and 
without waiving the procedural defects of Amplify’s untimely protest letter, I stand by the 
Department’s original contract award and deny your client’s protest. 

Istation is the best reading diagnostic tool for North Carolina educators, students, and 
parents. 


Sincerely, 


Mark R. Johnson 

North Carolina Superintendent of 

Public Instruction 
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Attachment A 


Overview 

State law requires North Carolina to have one reading diagnostic tool for every K-3 class 
and required the Department to evaluate diagnostic tools and select one to be implemented 
beginning with the 2019-2020 school year. 

“The State Superintendent shall fonn and supervise an Evaluation Panel to review the 
proposals received pursuant to the RFP issued in accordance with subsection (b) of this section.” 
Section 7.27(c) of S.L. 2017-57, as amended by Section 2.6 of S.L. 2017-197. This law also 
required that “[t]he Evaluation Panel shall be composed of persons employed within the 
Department of Public Instruction.” Id. 

“In determining which vendor to select, the Evaluation Panel shall consider, at a 
minimum, all of the following factors: (1) the time required to conduct formative and diagnostic 
assessments with the intention of minimizing the impact on instructional time; (2) the level of 
integration of assessment results with instructional support for teachers and students; (3) the 
timeliness in reporting assessment results to teachers and administrators; and (4) the ability to 
provide timely assessment results to parents and guardians.” Id. 

As directed by the General Assembly in Section 7.27(c) of S.L. 2017-57, as amended by 
Section 2.6 of S.L. 2017-197, “the Evaluation Panel, with the approval of the State 
Superintendent, shall select one vendor to provide the assessment instrument or instruments for 
the 2019-20 school year.” For clarity, any recommendation from the evaluation committee would 
also require my approval to become the selected vendor, which underscored my obligation, and 
enhanced my ability, as State Superintendent to ensure a fair and objective process. 

The Department, the State Board, and I fulfilled our statutory duties. In doing so, we 
consulted with the Department of Information Technology (“DIT”), the agency ultimately in 
charge of approving information technology (“IT”) procurement awards. The Department 
regularly consults with DIT to ensure the Department adheres to North Carolina’s procurement 
laws, rules, and policies. 
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Attachment B 


Read to Achieve Diagnostic Tool RFPs 

1. Introduction 

The procurement process for the Read to Achieve diagnostic tool faced unexpected 
challenges from within the Department. The process had to be cancelled and restarted twice to 
ensure fairness and objectivity, while also ensuring the Department and the State Board adhered 
to all state procurement laws, rules, and policies. 

At the start of the process in 2017, evaluation committee members promised to prioritize 
fairness and objectivity over improper biases; going so far as to sign non-disclosure agreements 
and statements affirming that they had disclosed all possible conflicts of interest. Unfortunately, 
a small number of career Department employees did not honor these commitments. 

Each evaluation committee member signed a Confidentiality Agreement. (See Exhibit A). 
Each member also signed a statement indicating that he or she had no conflict of interest. (See 
Exhibit B). 

The first RFP (“RFP-1”) was issued on December 6, 2017 and publicly opened on 
January 11, 2018. RFP-1 was cancelled on March 14, 2018 due to a lack of competition and 
errors in specifications. Even if RFP-1 had not been cancelled for this reason, the Department 
would have had to cancel RFP-1 due to an undisclosed conflict of interest. 

During the evaluation phase of RFP-1, the Department was made aware that a voting 
member of the evaluation committee, who was also one of the two business owners responsible 
for managing the process, did not disclose that she had previously been employed by your client, 
Amplify. She was paid by Amplify immediately preceding her employment with the Department. 
The Department was made aware of this conflict by a whistleblower who is a career school 
district employee. 

The second RFP (“RFP-2”) was issued on September 6, 2018 and publicly opened on 
October 2, 2018. RFP-2 was cancelled on March 21, 2019. RFP-2 had many legal and fairness 
issues (detailed below) that tainted the process, and the evaluation committee did not reach the 
consensus required by procurement rules to recommend one of the four diagnostic tools to me. 
The Department discovered improper and unfair procedures were used during the evaluation and 
that a voting member of the evaluation committee breached the Confidentiality Agreement by 
disclosing confidential details about the procurement process. 

2. RFP-2 Cancellation 

An agency can cancel an RFP when there are “circumstances that prevent detennination 
of. . . the best value offer; or [there is] any determination that rejection would be in the best 
interest of the State.” 09 NCAC 06B .0401. The RFP-2 evaluation process was plagued by 
multiple issues, resulting in the Department recommending to DIT that the RFP be cancelled. 
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Among these issues: 


a. breach of confidentiality agreement; 

b. unfair and improper ranking procedures; 

c. misstatements of fact leading to improper rankings; 

d. improper weights leading to improper rankings; and 

e. lack of consensus and unclear basis for comparison 

a. Breach of Confidentiality Agreement During RFP-2 

An anonymous whistleblower provided the Department with a screenshot of a text 
message chain between two fonner Department employees showing that a voting member of the 
evaluation committee breached the Confidentiality Agreement. 

Pursuant to 09 NCAC 06B .0103, the Confidentiality Agreement provides that “. . . all 
infonnation and documentation (verbal and written) relative to development of a contractual 
document is deemed ‘confidential’ and shall remain confidential until successful completion of 
the procurement process. Therefore, Evaluation Committee Members are required to keep all 
comments, discussions, and documentation confidential. . . .” (See Exhibit A). “If it is 
discovered that there has been a breach of confidentiality by a member of this Committee . . . 
[t]he solicitation may be cancelled . . . .” (See Exhibit A). 

During the text exchange, dated January 8, 2019, which is the same day as a meeting of 
the evaluation committee, one of the texting participants stated: “Well, just got off another 
marathon call with [employee’s initials redacted]. 1 hour 45 minutes all about RFP what a 
mess!” (See Exhibit C). 

The text messages describe who was on the evaluation committee and how each member 
voted during the meeting. The texting participant editorialized that some evaluation committee 
members “voted for children” while others “voted for helping teachers” so as to “appease lazy 
ass teachers.” (See Exhibit C). 

The Department still has an on-going investigation into this matter and the extent of the 

breach. 


b. Unfair and Improper Procedures Corrupted the Evaluation Committee’s 
Process in RFP-2 

The text exchange discussed above added to the concerns the Department and I already 
had about improper bias jeopardizing the legality and impartiality of RFP-2. 

The ranking process for RFP-2 was deficient. During the November 2018 meeting, the 
evaluation committee members were directed to vote “yes,” “no,” or “maybe” on each 
specification. Members were directed to vote “maybe” if clarification was needed. (See Exhibit 
D). 

However, the evaluation committee decided to rank vendors during the draft specification 
assessment without clarifying any specifications that received “maybe” votes. Instead, “maybe” 



votes were treated like “no” votes, and only “yes” votes were awarded points for ranking 
purposes. 

A proper clarification process would have yielded more “yes” votes for vendors. The 
Department discovered many specifications that received “maybe” votes (and counted as “no” 
votes) should have received “yes” votes. 

A clarification process should have occurred prior to any ranking by the evaluation 
committee. Vendors should have been receiving credit in this ranking process but were not, 
which corrupted the outcome. Further, misstatements of fact resulted in improper “no” votes and 
“maybe” votes (which counted as “no” votes) for Istation. 

c. Misstatements of Fact 

A PowerPoint presented to me in December 2018 contained multiple misstatements of 
fact and omissions of important vendor information. The presentation also had rankings of 
vendors based on these misstatements and omissions. 

As documented in the released public records, multiple misstatements of fact were put 
forth by members of the evaluation committee during the RFP-2 meetings. Many of these 
misstatements, such as those regarding dyslexia screening, required subsequent clarification and 
correction. 

The numerous misstatements of fact and biased infonnation found in the December 2018 
PowerPoint presentation were never corrected on that document because the presentation was 
never delivered publicly. 

The voting members of the evaluation committee for RFP-2 followed guidance from 
whomever they considered to be a given topic’s subject matter expert on the evaluation 
committee. Unfortunately, in multiple cases, the opinion voiced by a subject matter expert was 
not grounded in fact. Members of the evaluation committee, whether intentionally or not, were 
putting forth untrue statements about vendor products, but these falsehoods were treated as facts 
by the rest of the evaluation committee. Consequently, RFP-2 voting and rankings were based on 
misstatements of fact and incorrect data. 

The Department will likely never know the full extent of the number of points improperly 
denied to Istation during the RFP-2 process due to these misstatements of fact. At best, this was 
an unprofessional and unfair process. At worst, this allowed for improper bias on the part of an 
evaluation committee member to skew the results. This jeopardized the objectivity and fairness 
of the entire RFP-2 process. 

Even if the Department ignored these issues, the evaluation committee for RFP-2 
attempted but failed to reach a consensus, as per procurement guidelines, on a recommendation. 
Then the Department learned that a committee member had broken the Confidentiality 
Agreement. The related text exchange, unfortunately, featured participants accusing that certain 
voting members would “vote for children” while others were voting to “appease lazy ass 
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teachers.” This is disappointing evidence that misstatements of fact could have been intentional 
forms of improper bias. 

The following points, a non-exhaustive list, are key examples of misstatements of fact 
that led to improper votes. 

Misstatements of Fact = Improper Votes = Flawed Results 

Misstatement (evaluation committee opinion (summarized in December 2018 PowerPoint)): 
Istation is not a valid diagnostic assessment (based on 10 maybe votes, 1 no vote). 

Fact: 

Istation is a valid and reliable diagnostic literacy assessment and is used in all 50 states at 
the campus, district, or state level. It has been implemented statewide in eight states and is an 
approved K-3 screener in five additional states. The ISIP ER technical manual provides research 
underlying the assessment and includes results of research conducted on ISIP ER over multiple 
years showing strong concurrent and construct validity. ISIP ER reliability results for overall 
reading range from .927 to .970, which well exceeds required reliability at .80 or validity at .60. 
Results of predictability studies show ISIP ER can predict with confidence students’ 
performance on End of Grade exams and is a stronger predictor than DIBELS ORF for Texas 
Achievement of Knowledge and Skills (TAKS). 

Misstatement (evaluation committee opinion (summarized)): Istation is not a valid dyslexia 
screener (based on 2 maybe votes, 9 no votes). 

Fact: 

Istation is a valid dyslexia screener and ISIP ER is already used as a screener of risk for 
dyslexia in two states per legislation requiring dyslexia screening. ISIP ER provides reliable, 
continuous assessment in each of the critical areas of reading, including development of 
phonemic awareness and alphabetic knowledge and skills. Therefore, ISIP ER can be a powerful 
aid in identifying students with dyslexia early in their development. In a telephonic interview of 
Dr. Joseph Torgesen, one of the top experts in the field of dyslexia, conducted on July 12, 2019, 
he stated: “I consider this to be as good a screener as any I know of.” 

Misstatement (evaluation committee opinion (summarized)): Istation does not meet the 
requirements for progress monitoring (based on 5 maybe votes, 5 no votes, 1 yes vote). 

Fact: 

Istation is a valid progress monitoring tool and is currently being used for progress 
monitoring in all 50 states at the campus, district, or state level. Istation meets the requirements 
as a strong progress monitoring tool because it: (a) is brief - the assessment may be given to 
students in a one-on-one setting or with an entire classroom in approximately 30 minutes, (b) is 
repeatable - numerous items are available for each subtest, making the subtests repeatable 
throughout the school year, (c) is sensitive to improvement over time - ISIP ER has been 
designated to automatically provide continuous measurement of student progress throughout the 
year in all critical areas of reading, (d) documents improvement over time with instantaneous 


10 



reports to show student progress, and (e) allows for off grade progress monitoring as the 
assessment will automatically adapt to student ability even if it is below the students current 
grade. 

Misstatement (evaluation committee opinion (summarized in December PowerPoint)): Istation 
parent reports require the teacher to manually fill in the report for each child (based on 6 
maybe votes, 2 no votes, 3 yes votes). 

Fact: 

Istation automatically generates reports for teachers and parents that are available in 
English and Spanish, including the Student Summary Report which tracks each student’s 
progress in critical areas being assessed with complete graphic and contextual analysis. Several 
user-friendly printable and customizable templates are also available that teachers can choose to 
use in addition to the automatically provided reports. Also, parents in North Carolina will have 
access to the online parent portal where they can access student level reports that show specific 
skills in which each student struggles, and it links directly to the appropriate lessons and 
resources. 

Misstatement (evaluation committee opinion (summarized)): Screen-based assessments are not 
developmentally appropriate for Kindergarteners and struggling learners (based on 4 maybe 
votes, 7 no votes). 

Fact: 

ISIP ER computer-adaptive assessment is developmentally appropriate due to the very 
nature of the way items are administered that are individually selected to be at the right level of 
difficulty for each child. The measures are presented in an engaging format, thus reducing many 
of the problems, such as distraction and lack of interest, that are sometimes encountered when 
young children, especially struggling learners, are assessed. The computer-adaptive format 
engages young students as sights and sounds come to life with animated game-like interaction 
for young readers. The use of technology in the classroom has increased and will continue to 
increase in the future with all students. In a recent study of kindergarten students, Putnam (2016) 
found the level of teacher literacy support with the use of Istation had a statistically significant 
effect on early literacy achievement, explaining 17.7% of the variance in group differences. As 
the use of technology becomes more prevalent, the relationship between technology and literacy 
is at the forefront of educational decisions. This study contributes to the growing evidence that 
“the teacher’s role in the classroom is strengthened by the introduction of new technologies,” and 
“technology supplements, not supplants the teacher in the classroom.” 


d. Improper Weights Given to Criteria 

Pursuant to the North Carolina Administrative Code and the DIT manual, an IT 
procurement process conducted by the Department must use both the source selection method 
and the evaluation method designated in the solicitation. RFP-2 directed the use of the “tradeoff’ 
source selection method and the use of the “ranking” evaluation method (see RFP-2 pp. 13-14). 
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Under the tradeoff source selection method, “the criteria are stated in relative order of 
importance in the bid” but the evaluation must consider the criteria “without weighting them.” 
Statewide IT Procurement Office Manual 2017, Section 12.7.2. The evaluation committee 
improperly added weight to certain criteria in the RFP-2 process even though the solicitation 
document required use of the tradeoff source selection method. (See Exhibit E). 

Each evaluation criterion should be given the same weight, and each bid receives a 
relative overall ranking. In the next phase, the ranking evaluation method, the evaluation 
committee is then allowed to adjust those rankings “up or down when considered with, or traded- 
off against, other non-price categories.” 09 NCAC 06B .0302(2)(c). 

The State emphasizes the importance of equal weight as the first part of the process 
followed by potential adjustments for non-price categories because a goal of State IT 
procurements is to drive best value principles when buying services on behalf of the citizens and 
taxpayers of North Carolina. A best value procurement is “a procurement process with the 
objective of reducing the total cost of ownership.” 09 NCAC 06A .0102(2). The “particular 
procurement methods used [in a best value procurement] are selected so as to result in the best 
value for the State in terms of the function to be perfonned or delivered.” Id. 

Procurement laws and policies, when followed, drive the best value by having equal 
weight for cost at first while allowing an evaluation committee to later adjust for other factors if 
necessary. If multiple vendors can provide products, which meet the requirements of North 
Carolina law, an agency must consider cost in proper stewardship of the taxpayer dollars 
entrusted to it. This process allows agencies to procure products and services that are the best 
value to the State and its people. 

The RFP-2 evaluation committee improperly added weights to the scoring that prevented 
a best value procurement process. Because of this violation combined with the unfair and 
improper procedures described above, even a worthy product that was offered at no cost could 
have scored lower than Amplify. 

e. A Flawed Ranking, Not a Recommendation 

Even though, as discussed, there was improper bias, flawed procedures, lack of fairness, 
misstatements of fact, and other issues from RFP-2 that jeopardized the results, the evaluation 
committee did bring both Amplify and Istation to my attention as the top-ranked vendors. 
However, to reach an agreement to recommend one tool to me, the evaluation committee had to 
reach a consensus. The only vote in RFP-2 occurred in January 2019 - it was a vote on which 
vendor(s) to continue negotiating with - and did not result in consensus, so there was no 
recommendation. 

f. Cancellation of RFP-2 

With guidance from DIT, RFP-2 was properly cancelled. 
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Attachment C 


Negotiations Process 


1. Introduction 

The Department, after consulting with the State Board and DIT and properly cancelling 
RFP-2, entered the negotiations process pursuant to 09 NCAC 06B .0316. An important aspect 
of the negotiations process is that “[negotiations shall not materially alter the intent or scope of 
the original solicitation document.” 09 NCAC 06B .0316(e). The Department did not materially 
alter the intent or scope of the original solicitation document. 

2. Intent of the Original Solicitation 

The intent of the original solicitation document was to effectuate the goal of the Read to 
Achieve legislation. RFP-2 stated that “[t]he purpose of this RFP and any resulting contract 
award is to solicit offers for Read to Achieve Diagnostic Software ... to meet NCDPI’s 
obligations under state laws, which can be found at N.C.G.S. 115C-83.1 et. seq.” (see RFP-2 p. 
6). The solicitation was intended to “ensure that difficulty with reading development is identified 
as early as possible; students receive appropriate instructional and support services to address 
difficulty with reading development and to remediate reading deficiencies; and each student and 
his or her parent or guardian be continuously informed of the student’s academic needs and 
progress.” N.C. Gen. Stat. § 115C-83.2 (2018). The intent of the negotiations process was to 
effectuate the goal of the Read to Achieve legislation. The intent did not materially change. 

3. Scope of the Original Solicitation 

In RFP-2, the Department listed six criteria for evaluation. In the negotiations process, 
with guidance from DIT, the Department reworded certain criteria for brevity and clarity and 
removed two criteria that had already been fully addressed. The scope did not materially change. 
The negotiations process, however, did correct a flaw in RFP-2 by removing the weights from the 
scoring of the criteria. 


The chart below shows how the changes were not material alterations. 


RFP-2 Criteria 

Negotiations Criteria 

• Substantial Conformity to Solicitation 
Specifications 

• Reworded as “Formative & Diagnostic 
Assessment” for clarity* 

• RFP Desired Specification 

• Reworded as “Personalized Learning” for 
clarity** 

• Proof of Concept/Demonstration 

• Removed since this part of the process was 
properly completed by the evaluation 
committee during RFP-2 and did not need 
repeating 

• Vendor Cost Proposal 

• Vendor Cost Proposal 

• Vendor Relevant Experience and Reference 
Checks 

• Removed since this part of the process was 
properly completed by the evaluation 
committee during RFP-2 and did not need 
repeating 

• Vendor Financial Stability 

• Vendor Financial Stability 
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*As you state in the supplemental letter, “the whole purpose of the RFP is to procure ‘valid, reliable, formative and 
diagnostic reading assessments.’” (See RFP-2 p. 6). 

**ln RFP-2, the Department had included personalized learning as a substantive desired specification. Personalized 
Learning is a strategic direction of the State Board and clarification was added through this rewording in the 
negotiations process. 

4. Amplify’s Claims in the Supplemental Letter 

The theories purported in your protest are incorrect. The scope of the negotiations did not 
materially change from those of RFP-2. The Department did not change the rules of the 
procurement. Your claim that the Department “threw out” criteria is wrong. With guidance from 
DIT, certain criteria were removed because they were already satisfied for the evaluation 
committee and others were reworded for clarity. 

Properly following IT procurement policies, as discussed above, the criteria were not 
weighted during the negotiations process and were equal as a result. References and product 
demonstrations for Amplify and Istation were both properly completed and satisfactory. Being 
unweighted and equal, the vendors were equal for these criteria. 

With the guidance of DIT, the Department properly moved the cost proposal and vendor 
financial stability, both criteria from RFP-2, higher in priority without materially changing the 
scope. This did not have an impact on the outcome, though, because no adjustments were made 
by the evaluation committee based on the order of priorities. 

All the criteria were unweighted and equal. Amplify and Istation were both satisfactory 
for vendor financial stability. Being unweighted and equal, the vendors were equal for this 
criterion. Istation’s cost proposal provided during the negotiations had more strengths and fewer 
weaknesses than Amplify’s cost proposals provided during the negotiations. 

When given all the facts and the strengths and weaknesses for all criteria, the evaluation 
committee for the negotiations process unanimously agreed to continue further negotiations with 
Istation and recommended awarding to Istation. The evaluation committee had no need to adjust 
for other factors. 

5. Cost and Value 

As discussed above, the State emphasizes the importance of equal weight first. This can 
potentially be followed by adjustments for non-price categories because a goal of State IT 
procurements is to drive best value principles when buying services on behalf of the citizens and 
taxpayers of North Carolina. A best value procurement is “a procurement process with the 
objective of reducing the total cost of ownership.” 09 NCAC 06A .0102(2). The “particular 
procurement methods used [in a best value procurement] are selected so as to result in the best 
value for the State in terms of the function to be performed or delivered.” Id. 

Procurement laws and policies, when followed, drive the best value by having equal 
weight for cost at first while allowing an evaluation committee to later adjust for other factors if 
necessary. 

Cost was a significant factor in RFP-2. If multiple vendors can provide reading diagnostic 
tools that each meets the requirements of North Carolina law, the Department must consider cost 
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in proper stewardship of the taxpayer dollars entrusted to it. Per procurement policies, this 
maximizes value for the citizens of North Carolina. The Department made clear to Amplify and 
Istation that cost would continue to be a significant factor in the negotiations process. To 
emphasize the importance of cost in the process, and with the guidance of DIT, cost was given a 
high priority in the Request for Negotiation communication sent to both vendors. 

To maximize fairness during the negotiations process, the Department gave each vendor 
the opportunity to improve their price offered to the State. Amplify offered a reduced bid of $3.8 
million annually (plus professional development costs) compared to its bid of $6.1 million 
annually (plus professional development costs) in RFP-2. This bid was still objectively higher 
than Istation’s bid of $2.8 million annually (plus professional development costs). 

Though cost was given a high priority, the evaluation committee for the negotiations 
process did not use such priority in its detennination. Every criterion was unweighted and equal. 
The evaluation committee made its recommendation based on evaluations of the unweighted 
criteria and had no need to adjust for other factors. 

Your claims in the protest are wrong. Also, in your protest letter dated June 24, 2019, you 
take issue with a newspaper article that reported Istation’s contract amount for the contract award 
and Amplify’s current contract amount. The Department answered the questions asked by the 
reporter. While we cannot control what a reporter decides to compare, you use the opportunity to 
point out that Amplify’s bid was $3,755,560 a year, which you say represents more than a 40% 
reduction from prior years. 

In the next paragraph, however, you claim that Amplify’s tool offers more than Istation, 
specifically, the TRC. You do not properly qualify this argument with the fact that the additional 
offerings would cost more than the amount of Amplify’s bid. Amplify’s bid referenced in the 
protest letter was $3.8 million (plus over $500,000 for professional development) and did not 
include the TRC. Amplify also proposed an alternate bid, which included the TRC and added 
Amplify Reading National Edition, along with additional professional development. This bid 
was for $10.1 million in the first year (plus $1.2 million for professional development). 

A top priority of any IT procurement is to find the best value for the State in terms of the 
functions to be perfonned or delivered. After a fair and unbiased process, Istation proved to be 
that best value to the State. 

6. Recommendation to and Approval by the State Superintendent and the State Board 

of Education 

The evaluation committee for the negotiations process presented to me its unanimous 
recommendation to select Istation, and I approved the recommendation to select Istation. (See 
Exhibit F). Department staff and I then presented the recommendation to the State Board, and the 
State Board unanimously approved the recommendation to select Istation. 
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Attachment D 


Istation 


1. Introduction 

Throughout this protest for Amplify, you make incorrect claims about Istation to argue in 
favor of Amplify’s request. Istation has already been used successfully in other states and more 
than satisfies the requirements to be the reading diagnostic tool for North Carolina. Contrary to 
your incorrect claims: 

• Istation satisfied the requirements of the procurement, 

• Istation meets the requirements of law, 

• Istation predicts students who are at risk of a reading difficulty, 

• Istation screens for dyslexia, 

• Istation is developmental^ appropriate, 

• Istation assesses required measures, 

• Istation is effective at measuring a student’s progress, 

• Istation allows for students to interact with and read aloud to teachers, 

• Istation provides sufficient data to allow detenninations of short-term student progress, 

• Istation accurately and reliably detennines which students will be reading on grade level 
by the end of the school year, and 

• Istation is already on track for successful implementation in North Carolina. 

2. Istation’s Use in Other States 

Istation has been adopted for statewide implementation in eight states: Texas, Florida, 
Colorado, Kansas, New Mexico, Idaho, Arkansas, and North Carolina. In addition to the eight 
statewide implementations, Istation has been approved by state departments as a K-3 literacy 
screener in the following states: Massachusetts, Mississippi, Michigan, Ohio, and Oklahoma. 

Idaho 


The State Department of Education in Idaho recently released preliminary spring reading 
scores on July 1. Scores went up across the board from fall to spring. 

“The preliminary results from this first year of the new, comprehensive Idaho Reading 
Indicator are promising,” state superintendent Sherri Ybarra said in a news release. “The 
improvements were considerable. In each grade, the number of students reading at grade level 
improved by at least 12 percentage points. In first grade, the improvement approached 24 
percentage points. Put another way, more than 14,500 students who scored below grade level on 
the fall IRI advanced to grade level by spring.” 


Kevin Richert, What the New Reading Scores Say - And What They Don’t Say, Idaho Ed. 
News (July 11, 2019), https://www.idahoednews.org/news/what-the-new-reading-scores-say- 
and-what-they-dont-say/. 
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Florida 


Based on a recent report from Florida, results are promising for third grade students who 
have used Istation, as districts partnering with Istation indicate higher growth for third grade 
standardized testing in May. In the districts of Alachua, Monroe, Pinellas, and Polk, third grade 
scores “averaged a growth rate of 5.58% over the course of the FSA 2015-2019 testing results. 
This is over 2% more than average growth rate of districts that have not partnered with Istation. 
This average growth rate comes out to 3.4% over the same span of years.” 

“Istation helps raise the potential for growth in each student by allowing both teachers 
and students to monitor student growth, and by providing individualized education plans for each 
student based off of needs. These plans are scripted so a new teacher can easily use them, or if 
there is a seasoned educator using the technology they can smoothly add their own material to 
the personalized lesson so that each student gets the education they need rather than the 
traditional “one-size-fits-all” approach. “I love that it (Istation) tiers my students and provides 
individualized interventions that I can use to help my struggling students,” said Jessica 
Lockwood, a third-grade teacher at Poinciana Elementary School on Key West.” 

Online Staff, Higher Growth Reported for Third Grade Standardized Testing In Florida Districts 
Partnering With Istation, The Boca Raton Trib. (June 18, 2019), 

http://www.bocaratontribune.com/bocaratonnews/2019/06/higher-growth-reported-third-grade- 

standardized-testing-florida-districts-partnering-istation/. 

Arkansas 

A recent press release out of Arkansas speaks to reliability and validity of using Istation 
as a K-3 assessment. “Up to 40% of Arkansas's public kindergarten, first and second graders 
need additional support to achieve at a desired level in reading. That’s based on the 2019 results 
from the three different nationally standardized tests from which school districts can choose to 
give to pupils to gauge their achievements in reading and math. The state-authorized tests — 
Istation, Renaissance, and NWEA — produced similar results, which indicates the tests are 
reliable measures,” said the state’s director of assessment, Hope Worsham. 

Cynthia Howell, Education Notebook, Arkansas Democrat Gazette (July 15, 2019), 
https://www.arkansasonline.com/news/2019/jul/15/english-reading-scores-stir-queries-tes-l/. 

New Mexico 

New Mexico awarded Istation with a statewide contract in 2016 and faced a four-week 
timeline for implementation. Cloudcroft Municipal Schools K-8 Principal and DTC shared the 
following comments from her faculty and staff: 

“This experience has changed our minds about Istation. We weren’t a fan [sic] because of 
the quick turnaround we had to do. But the people with Istation have been very helpful and so 
nice so that was not as scary as we thought it would be. My teachers loved how quickly they got 
their data and access to intervention for students” (response on file with the Cloudcroft 
Municipal Schools). 
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"... I have an autistic student in third grade who is very negative, and it is hard to test 
because he refuses. He finished the entire test without redirecting from us. And even asked after 
the test when he could do it again. At one point he was belly laughing! That just doesn’t happen 
with him on a test or a computer.” Id. 

“You personally relieved my personal concerns for utilizing the program with five-year 
old children! I'm on board now to support and encourage our teachers to give this a try . ...” Id. 

“I am so relieved to say I really think we will love this new program! The Istation staff 
are all so very passionate and most of them are former teachers who used this in their own 
classrooms!” Id. 

“I feel very much at ease with it, and the other administrators I talked with were also very 
relieved and optimistic.” Id. 

“I have really been impressed with your company's responsiveness. I have dealt with 
support twice, and they were incredible.” Id. 

Kansas 

Kansas approved the entire program for Kansas Reading Success in 2015. A fifth grade 
teacher at Chanute Elementary School in Kansas shared how Istation has impacted her students’ 
learning, her teaching, and her classroom. Though Kansas differs from North Carolina because 
all schools use the curriculum, her comments about reports and simplicity for gathering data 
provide an important example of how teachers quickly gain insight into using data to inform 
instruction. 

“I love the reports! They give so much infonnation on exactly what the students are 
getting and what they aren’t. For example, I had one Tier 3 student who was doing great on 
Istation. With each passing week, her scores were improving. When the monthly assessment was 
pushed in, however, she tanked it. I was able to drill in and see exactly which questions she had 
missed. After the analysis and a conversation with her, it was apparent she bombed it on purpose; 
she was nervous about moving up to a harder level. After much encouragement she is back on 
track and moving forward!” 

“I love that it is pushed in each calendar month. It helps keep the kids motivated and is so 
easy for the teacher. If a student is absent the day of the assessment, it’s just there the next time 
they log on ... so easy on the teacher! Also, the reports generated from the assessments are 
amazingly data rich. I feel that it is meaningful data that drives both instruction and intervention. 
My Professional Learning Community (PLC) is considering using the Istation assessments as our 
common assessment next year to track student growth.” 

Lindsey Joyce, Year One With Istation: A Grade 5 Teacher Reflects, istation blog (May 4, 
2016), https://blog.istation.com/year-one-with-istation-a-grade-5-teacher-reflects. 
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3. Istation Feedback from North Carolina 

Eric Davis, Chairman of the State Board, told me that he discussed the State Board’s 
decision with education experts he trusts, and they told him they consider Istation one of the top 
diagnostic tools available (second only to another vendor that applied but did not meet all the 
criteria of the Read to Achieve law (not Amplify’s tool)). 

Multiple meetings of the State Board have occurred since the contract award to discuss 
Istation and its implementation in the 2019-2020 school year. The State Board has been 
supportive of the contract award and has not budged from its decision. When discussing the 
training schedule for Istation in a board meeting on June 28, 2019, State Board member Amy 
White said: “This is not a discussion about whether we’re going back to mCLASS.” 

T. Keung Hui, After outcry from educators, NC will delay use of computer-based reading test, 
The News & Observer (June 29, 2019, updated June 29, 2019), 
https://www.newsobserver.com/news/politics-government/article232045842.html. 

At the same meeting, Amy White said: “I think having something new is always hard. 
Moving from one platform to another is always hard. But I think once they see it, touch it, feel it, 
they will realize that [Istation’s] going to be something very positive.” 

Board member Dr. Olivia Oxendine said: “This is more of an instructional feature 
advantage that I see in Istation. The other thing is that I go back to the year 2013, definitely 
2014-15, and there was an incredible angst around the state among classroom teachers and 
curriculum specialists and superintendents about the amount of time taken away from sheer, pure 
instruction so that so much could be done with assessment, the assessment of students. So, we 
received during that period of time, probably about 18 months to two years, a barrage of emails 
from superintendents and teachers about not enough, not sufficient time given to the direct 
instruction of reading. So, I believe that we are going to have the opportunity to allow teachers to 
do what is called teaching reading and acknowledging and teaching to the science of reading, 
which is music to my ears.” 

Jack Hoke, the Executive Director of the NC School Superintendents Association, 
emailed me and told me that he attended a session in Asheville with Istation. He said it appeared 
to be an excellent tool. 

The number of positive responses from North Carolina educators continues to grow as 
more and more attend Istation trainings. Please see the current compilation of responses collected 
after educators completed face-to-face training with Istation. (See Exhibit G). 
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4. Amplify’s Claims are Incorrect 


a. Istation Meets the Requirements of Law 

Before this process, Istation was approved by the State Board as an assessment for 
proficiency in third grade reading under the Read to Achieve law and was used by others in 
North Carolina at their own expense. 

Istation meets all the criteria of the Read to Achieve law (G.S. § 115C-83.6): 

i. Istation serves as a valid, reliable, formative, and diagnostic reading assessment for 
students in kindergarten, first, second, and third grades. 

ii. Istation provides immediate data that can be used with the Education Value-Added 
Assessment System (EVAAS) to analyze student data to identify root causes for 
difficulty with reading development and to detennine actions to address them. 

iii. Istation provides formative and diagnostic assessments and resultant instructional 
supports and services to address oral language, phonological and phonemic awareness 
phonics, vocabulary, fluency, and comprehension using developmentally appropriate 
practices. 

iv. Istation is administered by computer or some other electronic device as allowed by law. 

b. Istation Adequately Predicts Students Who Are at Risk of a Reading Difficulty 

The authors, Dr. Joseph Torgesen and Dr. Patricia Mathes took great care in constructing 
the ISIP ER item pool, basing the types and content on contemporary findings in early reading 
research. Istation’s ISIP ER measures are designed to be used for Universal Screening 
(benchmarks three times a year) and Progress Monitoring (monthly). The benchmarks and 
progress monitoring assessments are designed to test the same skills in the same fonnat and with 
the same consistent and comparable results. While the questions will change for the individual 
student, the subtest, the fonnat, and overall score composition will remain consistent between 
progress monitoring and benchmarks to ensure consistency in instructional design and 
instructional decisions between benchmark months. 

The advanced technology used in Istation’s adaptive fonnat provides data to help 
teachers determine which students are at risk for reading difficulties and those who have missed 
critical foundational skills that are typically not assessed at later grade levels. For each student, 
the tasks adapt to lower or higher-level skills - meeting the student where he or she is on the 
student’s reading journey - and produces a reliable estimate of their ability. 

Reliability and validity are two important qualities of measurement data. Reliability can 
be thought of as consistency and validity can be thought of as accuracy. The ISIP Early Reading 
Technical Report (c 2011; 2016) describes a study conducted using ISIP Early Reading data 
collected from kindergarten through grade 3 in five elementary schools in north Texas. 

Regarding measures of reliability, data suggest consistently high levels of internal consistency in 
the subtest ability scores as well as the overall reading ability scores. ISIP ER also produces 
stable scores over time. Evidence of concurrent validity is found in the numerous strong, positive 
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relationships to external measures of reading constructs. Data suggest mostly large to very large 
criterion validity with scores from well-known measures of reading constructs, such as CTOPP, 
GORT-4, PPVT-III, TOWRE, WJ-II ACH, WLPB-R, and WIAT-II, as well as with TPRI and 
ITBS. In addition, classification accuracy and area under the curve estimates for third grade 
accountability measures meet generally accepted standards. 

Furthermore, predictive validity results show that ISIP Overall Reading is a stronger 
predictor than DIBELS ORF for state mandated Texas Assessment of Knowledge and Skills 
(TAKS) Reading, using scores from 1 to 5 months prior. Taken together, the evidence supports 
that ISIP ER produces reliable and valid data for measuring key areas of reading development 
such as phonemic awareness, alphabetic knowledge, vocabulary, and reading comprehension, as 
well as overall reading ability. 

Additionally, Campbell, L. O., Lambie, G. W., and Sutter, C. conducted a study to 
examine third grade ISIP-ER scores and Florida Standards Assessment English Language Arts 
(FSA-ELA) scale scores. Data were collected during the 2016-2017 school year from five public 
school districts in Florida. Researchers used a simple linear regression analysis to examine the 
correlation of Overall Reading Ability and Reading Comprehension to the FSA-ELA scores. 
Results of the study indicated ISIP ER scaled scores had a strong correlate to FSA-ELA scores 
and predictability bands were computed to identify cut scores to predict all achievement levels. 
This study provides evidence that Istation’s reading assessment can predict the FSA-ELA 
statewide examination scores for all achievement levels among third grade students. 

Campbell et. al., (2018). Measuring the Predictability oflstation Indicators of Progress Early 
Reading (ISIP-ER) Scores on Florida Standards Assessment Scores. University of Central 
Florida, www.ucf.edu/mirc. 

Recently, Campbell, Sutter, Lambie, and Tinstman (2019) conducted a study to measure 
the predictability of Istation’s Indicators of Progress (ISIP) Early Reading Scores on Renaissance 
STAR Reading. Researchers examined kindergarten, first, and second grade ISIP-ER scores 
(Overall Reading Ability and Reading Comprehension) in one county in the state of Florida. 

Data were collected during the 2017-2018 school year and a simple linear regression analysis 
was conducted to determine the correlation of the STAR scaled scores and the ISIP-ER overall 
reading scores. Results indicated ISIP-ER scaled scores had strong correlate to the STAR scores 
and predictability bands were computed to identify cut scores for accurate prediction for all 
achievement levels. This study provides evidence that ISIP Reading cut scores can predict the 
STAR statewide examination scores for all achievement levels among first and second grade 
students using computer-adaptive testing. 

Campbell et. al, (2019). Measuring the predictability oflstation Indicators of Progress Early 
Reading (ISIP-ER) scores on Renaissance STAR Reading scores. University of Central Florida, 
www.ucf.edu/mirc. 
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c. Istation Screens for Dyslexia and Meets the Statutory Requirements for 
Screening Dyslexia 

Istation is used in other states as a dyslexia screener for children. One of the authors of 
ISIP Early Reading (ISIP ER) was Dr. Joseph Torgesen, an Emeritus Professor of Psychology 
and Education at Florida State University. He is also the author, with Dr. Richard Wagner and 
Dr. Carol Rashotte, of two of the most widely used diagnostic tests for dyslexia. The 
Comprehensive Test of Phonological Processes (CTOPP) and the Test of Word Reading 
Efficiency (TOWRE). Dr. Torgesen was asked to establish the Florida Center for Reading 
Research (FCRR), which has been the premiere site for disseminating information about 
research-based practices related to literacy instruction and assessment for children since 2002. In 
the development of ISIP ER, several subtests were included that are often used to help in an 
initial screening for dyslexia. ISIP ER measures have high correlations with other measures that 
are used for dyslexia screening; therefore, ISIP ER measures can confidently be used to screen 
for dyslexia, particularly if they are given on a monthly basis. Additionally, Dr. Torgesen stated, 
“I consider this to be as good a screener as any that I know of.” 

Data related to student perfonnance in the areas that can be predictive red flags of 
dyslexia and difficulties in other areas of reading (e.g., phonemic awareness, letter sound-symbol 
recognition, letter knowledge) are provided by ISIP ER to identify student skill weaknesses, 
evaluate student intervention plans, discuss student perfonnance with administrators, and plan 
for parent-teacher conferences. From research, we know that early diagnosis of dyslexia is more 
likely when the problems occur in students who have strong abilities in other areas of language 
such as vocabulary. Istation measures listening comprehension of students in kindergarten as 
well as vocabulary K-3, thus allowing educators to gather assessment data for vocabulary, which 
has been an optional measure with mCLASS. 

Characteristics associated with reading difficulties are connected with spoken language. 
Difficulties in young children can be assessed through screenings of phonemic awareness and 
other phonological skills (Sousa, 2005). Additionally, Eden (2015) points out that “when 
appropriate intervention is applied early, it is not only more effective in younger children, but 
also increases the chances of sparing a child from the negative secondary consequences 
associated with reading failure. ISIP ER is a tool that can be used to screen for risk of dyslexia 
early in students’ academic careers.” 

Beginning with phonological processing, measures of phonological awareness tend to be 
the most predictive of success at early reading. As stated by Dr. Joseph Torgesen, “In young 
children, the most reliable indicators of risk for dyslexia are difficulties in the development of 
phonemic awareness and alphabetic knowledge and skills. The latter category includes letter- 
sound knowledge and alphabetic reading skills (ability to “sound out” unknown words). If 
difficulties in these areas are not addressed by powerful instruction early in a child’s 
development (kindergarten and first grade), the student typically does not progress in the ability 
to read text fluently, and this is an extremely reliable indicator of dyslexia by the end of first 
grade. Since the ISIP-ER provides reliable, continuous assessment in each of these critical areas, 
it can be a powerful aid in identifying students with dyslexia early in their development.” 
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It is important to note that ISIP ER does include measures that can be used to screen 
students for dyslexia, and Dr. Joseph Torgesen has confirmed that requiring any additional 
measures is not necessary to accurately identify students who are at risk for dyslexia. However, 
Istation has worked with North Carolina to identify a process that has been used in Arkansas to 
further help and flag students for further evaluation. Using the model that has been successful in 
Arkansas, educators in North Carolina could use results from an ISIP screening that would be 
gathered at a time other than a benchmark, including measures that are not typically administered 
at that time/grade level, in order to identify students who are in need of additional advanced 
screening. The initial screening can take effect as follows: 

• Kindergarten: Kindergarten students can be automatically screened upon login, at which 
point they can receive the additional subtest of alphabetic decoding. This subtest can be 
given later in the year in addition to the normal ISIP subtests for kindergarten: 
listening/language comprehension, phonological and phonemic awareness, letter 
knowledge, and vocabulary. 

• 1st Grade: 1st graders will automatically receive the relevant subtests based on the initial 
screening. These subtests include phonological and phonemic awareness, letter 
knowledge, vocabulary, alphabetic decoding, reading comprehension, and spelling. 

• 2nd Grade: 2nd grade students can be automatically screened upon login with the 
additional subtests of alphabetic decoding, letter knowledge, and phonological and 
phonemic awareness. These subtests would be in addition to the nonnal ISIP subtests for 
2nd graders: vocabulary, reading comprehension, spelling, and text fluency. 

• 3rd Grade: 3rd grade students can be automatically screened upon login with the 
additional subtests of alphabetic decoding, letter knowledge, and phonological and 
phonemic awareness. These subtests would be in addition to the nonnal ISIP subtests for 
3rd graders: vocabulary, reading comprehension, spelling, and text fluency. 

For clarity, while the Department and I desire these functions for our Read to Achieve 
diagnostic tool, and Istation provides these functions, your claim that Session Law 2017-127 
(HB 149) requires the Read to Achieve diagnostic tool to screen for dyslexia is not accurate. 
Session Law 2017-127 states that “it is the intent of the General Assembly that all students with 
specific learning disabilities, including dyslexia and dyscalculia, receive the necessary and 
appropriate screenings, assessments, and special education services to provide interventions for 
learning difficulties with language, reading, writing, and mathematics.” S.L. 2017-127, §1. 

The law goes on to state that “prior to the start of the 2017-2018 school year, local boards 
of education shall review the diagnostic tools and screening instruments used for dyslexia, 
dyscalculia, or other specific learning disabilities to ensure that they are age-appropriate and 
effective and shall determine if additional diagnostic and screening tools are needed.” Id., at §4. 
The law also requires the State Board to craft definitions related to dyslexia identification and to 
ensure the availability professional development opportunities related to identification of 
dyslexia. The law does not actually require the Read to Achieve tool to serve as a dyslexia 
screener for the state. Id., at §2-3. 
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Again, though, I agree that the Read to Achieve diagnostic tool serves as a good 
opportunity to provide local school districts with dyslexia screening and am pleased that Istation 
does so. 


d. Istation is Developmentally Appropriate 

One important piece of Istation is the modeling opportunity provided by the teacher for 
students before students interact with the assessment. The computer-adaptive nature of Istation 
ensures that each student, regardless of grade level, will be assessed in a way that identifies his 
or her individual proficiencies as well as challenges in reading. Data are immediately available 
through benchmarks and progress monitoring to differentiate instruction in order to address 
individual needs. Istation reports also provide for grouping of students by skill perfonnance for 
small group instruction. Recommendations for individual and small group instruction, as well as 
interventions are provided through Istation to support the teacher in data-based decision making 
and instructional planning. Teachers have access to a repository of resources, including lesson 
plans and instructional activities specific to each student’s needs. 

Dr. Joseph Torgesen states, “A computer-adaptive assessment like ISIP ER is 
developmentally appropriate because it administers items that are individually selected to be at 
the right level of difficulty for each child. In addition, because of its lively and engaging format, 
it reduces many of the problems, such as distraction and lack of interest, that are sometimes 
encountered when young children are assessed.” 

With Istation, teachers will assess their own students and obtain valid, reliable, and 
authentic data using technology tools and skills. For mCLASS: Reading 3D, there is variance in 
the level of teacher training that often yields unreliable data, especially as it relates to the 
components with more difficult scoring rules, such as NWF and PSF, and TRC. There are 
considerable concerns related to the subjectivity associated with administration of mCLASS 
Reading 3D, especially TRC measures. This is evident in the fact that teachers do not assess 
their own students at MOY and EOY. Instead, teachers across the state have assessed students in 
another classroom on their grade level to increase the objectivity in administration using 
mCLASS Reading 3D and obtain scores for students that are nonbiased and accurate. Even 
though students interact via technology with Istation, students are expected to read text just as 
students read text on Flex or any other mCLASS measure, but ISIP measures will adjust based 
on the student’s performance. 

With Istation, students absolutely can produce the sounds and read the words out loud or 
silently. The teacher will have the flexibility to monitor students based on individual need as they 
apply their knowledge to complete tasks to demonstrate proficiency on the critical skills in 
reading. In addition, K-12 students have Infonnation and Technology Essential Standards 
required by the state which include the following: Kindergarten: Use technology tools and skills 
to reinforce classroom concepts and activities. 

As documented in the ISIP Early Reading Technical Report: following a Computer 
Adaptive Testing (“CAT”) model, each item within the testing battery is assessed to detennine 
how well it discriminates ability among students and how difficult it actually is through a process 
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called Item Response Theory (IRT) work. Once item parameters have been determined, the CAT 
algorithm can be programmed. Using this sophisticated algorithm, the computer selects items 
based on each student’s performance, selecting easier items if previous items are missed and 
harder items if the student answers correctly. Through this process of selecting items based on 
student perfonnance, the computer generates “probes” that have higher reliability than those 
typically associated with alternate fonnats. This results in a better reflection of each student’s 
true ability. The process described above provides evidence that the true ability of students in K- 
3 will be established using the computer-based assessment model of ISIP ER Assessment. 

As it relates to the developmental appropriateness of using a computer-adaptive 
assessment like ISIP ER for young children, Dr. Joseph Torgesen stated, “There is no research 
that I am aware of to indicate that the small increases in time spent in front of a computer screen 
using the ISIP-ER constitute a risk for young children. In fact, monthly assessments using the 
ISIP-ER very likely involve an extremely small proportion of the total time young children spend 
in front of computer, or TV, or IPAD screens in our modern world.” 

Lastly, while Istation does meet the criteria as described above, in an attempt to support 
Amplify’s approach, your protest letter added inappropriate context for “developmentally 
appropriate” attached to the description of this RFP specification around observation-based 
practices. The actual specification is: “Describe how the proposed solution directly assesses 
reading and pre-reading behaviors to support students’ learning development at the various grade 
levels to inform instruction, including any observation-based practices if applicable.” 

e. Istation Assesses Required Measures 

As noted in the ISIP Early Reading Technical Report (c 2011; 2016), ISIP Early Reading 
follows a continuum of learning that, research indicates, is predictive of later reading success. 
Skills build upon skills, and the sequence of subtests builds upon prior subtests. As skills of 
lower-level difficulty are achieved, they are eliminated from the test battery and more advanced 
skills are added. Importantly, there is no other continuous progress monitoring assessment tool 
that measures vocabulary and comprehension accomplished through short tests, or “probes,” 
administered at least monthly, that samples critical areas that predict later performance. Id. 

ISIP Early Reading provides growth information in the five critical domains of early 
reading: phonemic awareness, alphabetic knowledge and skills, fluency, vocabulary, and 
comprehension. It is designed to (a) identify children at risk for reading difficulties, (b) provide 
automatic and continuous progress monitoring of skills that are predictors of later reading 
success, and (c) provide immediate and automatic linkage of assessment data to student learning 
needs, which facilitates differentiated instruction. Below is a grade level breakdown of how 
Istation ISIP ER measures assess all required measures plus listening comprehension, 
vocabulary, and spelling that are not assessed with mCLASS: Reading 3D. Id. 

Kindergarten (Subtests - Listening Comprehension, Phonemic Awareness, Letter Knowledge, 
Vocabulary): 

Kindergarten students require assessment of their growth in phonemic awareness, 
alphabetic knowledge and skills, and vocabulary. Fluency in letter names and sounds facilitates 
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spelling, but these skills are usually not developed sufficiently to assess spelling ability. Their 
reading skills are also rarely sufficiently developed to usefully assess reading fluency and 
reading comprehension. In general, research has shown that phonological awareness and letter 
sound knowledge in Kindergarten are predictive of Grade 1 outcomes. For children at risk of 
reading difficulty due to poverty or language background, vocabulary is critical to reading 
success (Foorman, Anthony, Seals, & Maouzaki, (2016); Snow et ah, 1998; Dickinson & Tabors, 
2001). Vocabulary assessments for Kindergarten students are mostly “tier 1” words and items to 
assess children’s knowledge of prepositions and verbs of varying tense, since these classes of 
words are particularly difficult for young children. Id. 

Grade 1 (Subtests - Phonemic Awareness, Letter Knowledge, Vocabulary, Alphabetic Decoding, 
Comprehension, Spelling, Connected Text Fluency, Oral Reading Fluency): 

It is important to continue to monitor students’ development of phonemic awareness and 
alphabetic knowledge and skill, because struggling students may continue to have difficulty in 
these areas. The development of accurate and fluent decoding skills should be monitored, since 
these foundational skills for reading accuracy undergo major development. Word recognition at 
the beginning of Grade 1 has been found to be predictive of Grade 1 outcomes. Spelling has also 
been found to be a predictor of oral reading fluency. Vocabulary growth is important in the 
development of reading comprehension. As soon as students can demonstrate the ability to read 
connected text with reasonable accuracy and understanding, reading fluency (timed reading with 
meaning) should be monitored. Continued growth in vocabulary should be assessed, as well as 
reading comprehension. Id. 

Grade 2 (Vocabulary, Comprehension, Spelling, Connected Text Fluency, Oral Reading 
Fluency): 

Word reading continues to be a strong predictor of Grade 2 outcomes, with reading 
fluency and comprehension becoming increasing important predictors. Second graders need 
continued monitoring of their decoding abilities, because struggling students may still have 
difficulty in this area. Reading fluency is critical through second grade, since students must make 
strong growth in this skill to maintain grade level reading proficiency. The development of 
reading comprehension is dependent on fluency and vocabulary. Sight vocabulary must grow 
rapidly in second grade to keep pace with expected reading outcomes. Thus, continued growth in 
Spelling, Vocabulary and Reading Comprehension should be measured. Id. 

Grade 3 (Vocabulary, Comprehension, Spelling, Connected Text Fluency, Oral Reading 
Fluency): 

Reading fluency and comprehension are strong predictors of Grade 3 outcomes. The 
primary dimensions of reading growth that should be measured in third grade are reading 
fluency, reading comprehension, spelling and vocabulary. Id. 

f. Istation is Effective at Measuring a Student’s Progress 

ISIP ER measures are designed to be used for Universal Screening (benchmarks three 
times a year) and Progress Monitoring (monthly). The benchmarks and progress monitoring 
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assessments are designed to test the same skills in the same format and with the same consistent 
and comparable results. While the questions will change for the individual student, the sub test, 
the fonnat, and overall score composition will remain consistent between progress monitoring 
and benchmarks to ensure consistency in instructional design and instructional decisions between 
benchmark months. 

As stated in the ISIP ER Technical Manual (c 2011; 2016), “With recent advances in 
Computer Adaptive Testing (“CAT”) and computer technology, it is now possible to create 
assessments that adjust to the actual ability of each child. Thus, CAT replaces the need to create 
parallel forms. Assessments built on CAT are sometimes referred to as “tailored tests” because 
the computer selects items for students based on their performance, thus tailoring the assessment 
to match the perfonnance abilities of the students. This also means that students who are 
achieving significantly above or below grade expectations can be assessed to more accurately 
reflect their true abilities.” 

Furthennore, the Technical Manual (c 2011; 2016) indicates, “There are many 
advantages of using a CAT model rather than a more traditional parallel fonns model, as is used 
in DIBELS. First, it is virtually impossible to create alternate forms of any assessment that are 
truly parallel. Thus, reliability from form to form will always be somewhat compromised. 
However, when using a CAT model, it is not necessary that each assessment be of identical 
difficulty to the previous and future assessments. Following a CAT model, each item within the 
testing battery is assessed to determine how well it discriminates ability among students and how 
difficult it actually is through a process called Item Response Theory (IRT) work. Once item 
parameters have been determined, the CAT algorithm can be programmed. The use of CAT 
algorithms also creates efficiencies in test administration. The adaptive item algorithm allows the 
computer to adjust item difficulty while the child is taking the test, quickly zeroing in on ability 
level. Thus, the use of CAT algorithms reduces the amount of time necessary to accurately 
detennine student ability.” 

The entire assessment battery for any assessment period requires 40 minutes or less. It is 
feasible to administer ISIP Early Reading assessments to an entire classroom, an entire school, 
and even an entire district in a single day, given adequate computer resources. Classroom and 
individual student results are immediately available to teachers, illustrating each student’s past 
and present performance and skill growth. Teachers are alerted when a particular student is not 
making adequate progress so that the instructional program can be modified before a pattern of 
failure becomes established. 

With the advanced computer technology, data are provided in real time to help teachers 
detennine which students are at risk for reading difficulties and those who have missed critical 
foundational skills that are typically not assessed at later grade levels. For each student, the 
computer adapts to lower or to higher level skills and produces a reliable estimate of their ability. 
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g. Istation Provides Opportunities for Students to Read Aloud to and Interact with 
their Teachers 

Students can read to and interact with teachers with Istation in place. The teacher can and 
very much should be highly involved in the reading and learning of a child. Teachers are 
encouraged to sit with the students as they read aloud and participate in learning activities. For 
example, in Istation’s Oral Reading Fluency subtest, the teacher or reading specialist is asked to 
listen to the child read aloud and identify key reading behaviors and skills. 

With Istation, students absolutely can produce the sounds and read the words out loud to 
teachers. The teacher will have the flexibility to monitor students based on individual needs as 
they apply their knowledge to complete tasks to demonstrate proficiency on the critical skills in 
reading. 

The purpose of K-3 fonnative assessments is to provide data pertaining to a student’s 
perfonnance level, not to substitute for teacher interaction. Teacher interaction is critical during 
instructional time, as research consistently demonstrates evidence-based instruction informed by 
meaningful data produces changes in student outcomes. Teachers who monitor their student’s 
progress and use the data to infonn instruction and decision making have higher student 
achievement. In fact, results from a study conducted by SERVE (2015) indicated teachers 
reported using many different measures to assess students prior to the introduction of mCLASS: 
Reading 3D and 96% of teachers indicated that they used fonnative assessment to infonn 
instruction before mCLASS: Reading 3D. 

Since a CAT like the ISIP-ER is efficient, and items are individually selected each time a 
student takes the test, it can be given frequently enough to produce reliable estimates of student 
growth from month to month. This is, of course, very important infonnation for the teacher, who 
can adjust teaching strategies, or teaching intensity, based on whether or not each individual 
student is growing in the critical reading skills assessed by the ISIP-ER. Valid, timely, and 
reliable assessment data gathered in an effective and efficient way provides more time for the 
teacher to interact with students as he or she engages in effective small group, targeted reading 
instruction. Gains in instructional time allow teachers to use small group instructional sessions to 
determine where individual children fall on the continuum of reading skills and provide 
systematic, explicit instruction to promote reading proficiency. 

Flex assessments in mCLASS are also computer-based and gather data through a device. 
Students are asked to read and stop to select a correct answer. Teachers have reported an 
appreciation for the option to use Flex due to the reduced time spent assessing their whole class. 
Istation takes this flexibility to another level in a way that still allows for audible phonemics to 
be sounded out by students, while saving more instructional time for both students and teachers. 

ISIP-ER can provide an accurate and efficient assessment of student ability because it 
uses advanced computer technology to select items that are “just right,” in terms of their level of 
difficulty, for each individual student. Thus, students do not waste time responding to test items 
that are “way too hard” or “way too easy.” The computer can also estimate when the student has 
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responded to enough items of the appropriate level of difficulty in order to produce a reliable 
estimate of their ability. As the Putnam (2016) study demonstrates, with the relationship between 
technology and literacy being at the forefront of many educational decisions, “the teacher’s role 
in the classroom is strengthened by the introduction of new technologies,” and “technology 
supplements, not supplants the teacher in the classroom.” 

h. North Carolina is Already Engaged in a Successful Implementation of Istation 

The Department and Istation have developed a statewide implementation schedule to 
have initial training for all educators complete before the start of school. With the ease of 
administering Istation in combination with the level of support offered in online modules, face- 
to-face training, webinars, and on demand links, successful implementation of Istation is well 
underway. The training and implementation plan include the following steps: 

• Begin using Istation at the start of the 2019-2020 school year (with delay in metrics 

described below) 

• Gather data during the fall to start becoming familiar with the assessments 

• Start using data to infonn instruction 

• Delay the use of data to measure growth for EVA AS until MOY benchmark; 

• Use first official benchmark in the winter (MOY) and end of year benchmark (EOY) for 

EVAAS purposes 

• Train all teachers by start of school with a schedule including: 

- In-person workshops (face-to-face regional-based half day train the trainer model for 
one campus leader per school and district leader/RtA Contact) 

- On-demand webinars (monthly technical and educator live webinar in Q&A 
environment recorded and available for on demand viewing) 

- Learning modules (monthly timely topics in released online modules/podcasts for 
educators to view brief course, complete quiz, get certificate of completion) 

- Technical assistance 

- Ongoing support from the K-3 Literacy team 
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Below is the “Getting Started with Istation Roll Out Plan,” which has the support of local 
superintendents and the State Board. 

Summer 2019 (June, July, August) 

• Istation enrollment and deployment activated in schools 

• Live webinars hosted 

• In-person regional trainings hosted 

• On-going implementation support 

• Districts can begin using Istation as early as July, but it will not count in metrics 

Fall 2019 (September, October, November) 

• Students take Istation’s ISIP assessment to begin to leam about the program 

• Additional in-person regional trainings hosted 

• Additional live webinars hosted 

• Fall is a “Getting Started” learning opportunity. Data will not feed into EVAAS 

Winter 2019-2020 (December, January, February) 

• Ongoing progress monitoring continues 

• Ongoing training continues 

• January: The first benchmark window for 2019-2020 opens for EVAAS purposes 
(MOY)* 

Spring 2020 (March, April, May) 

• Ongoing progress monitoring continues 

• Ongoing training continues 
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• May: End-of-year benchmark window for 2019-2020 opens for EVAAS purposes 
(EOY)* 

(*Traditional School Calendar) 

As of July 25, 2019, over 2,735 North Carolina educators and districts leaders have 
engaged in webinars and in-person trainings. Educators are already modeling ISIP measures with 
students in year-round schools. 

• 989 total attendees in tech webinars 

• 1,241 total attendees in live educational webinars 

• 505 total attendees in person face to face workshops 

On a survey following training, participants were asked to, “Rate how helpful you feel 
Istation’s ISIP assessment will be for you and your campus on a scale of 1-5 stars, with a 1 being 
not very helpful, and a 5 being very helpful.” 


Percent vs. On a scale of 1-5 stars, with a 1 being not very helpful, and a 5 
being very helpful, rate how helpful you feel Istation's ISIP assessment will be 
for you and your campus: 




To date, results indicate: 

32% = 5 stars, 38% = 4 stars, 23% = 3 stars, 6% = 2 stars, 2% = 1 star 

Additionally, technical integration of Istation and the Department has been a smooth 
process. Integration with PowerSchool has been seamless. Rostering occurs automatically with 
nightly updates so that any changes are applied instantaneously. Integration with NCEdCloud 
has been successful, and we already have teachers logging in successfully to Istation via 
NCEdCloud. 

• 114 districts/charters signed up through registration 

• 450 campuses rostered 

• 105,000 students rostered 

• 629 teacher logins 

i. Istation Satisfied the Requirements of the Procurement 

As discussed in this response, Istation satisfied all the requirements of the procurement. 

j. Istation’s ISIP-ORF was not Considered in the Contract Award 

This feature described by Amplify was not a factor in the contract award. 
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Attachment E 


Amplify 

In your protest letter, you state, “The agency must also consider factors including 
whether the vendor complies with industry standards, the vendor’s past perfonnance with the 
state, and the probability of the vendor providing the required service on time. 09 NCAC 
06B.0302(l)(f)(iii).” In responding to your letter with these discussions of Amplify’s tool, 
mCLASS, and Istation, I would like to take this opportunity to share concerns with Amplify’s 
product. 

a. Teachers’ Concerns with mCLASS 

A study was conducted by the SERVE Center at UNC Greensboro who contracted with 
the Department to report on initial Read to Achieve implementation. One major finding for the 
2014 Year One Report was, “implementation challenges identified by respondents that need 
attention, mostly related to administering benchmarking and progress monitoring assessments 
and a perception that reading assessments are taking too much time from instruction.” 

Given the results of 2014 report, the Department again contracted with the SERVE 
Center to conduct the current (2015) study to further examine teacher perceptions and 
assessment processes of mCLASS: Reading 3D in classrooms due to the questions from 
educators and the State Board as to how long the assessments in mCLASS: Reading 3D were 
taking, whether students lost too much instructional time or time spent on other educational 
activities, and whether the results of the formative assessments were being used to modify and 
tailor individual instruction. 

Researchers analyzed data from teacher logs, observations, and interviews of 100 
teachers from 37 randomly selected schools across the state and 6 schools that the Department 
recommended as exemplars, as well as results from 749 teachers’ responses to a survey in order 
to gain an understanding of the assessment process. Results indicated an overwhelming concern 
with the time involved with mCLASS: Reading 3D that is taken from providing quality 
instruction to students that improves reading skills. Nearly half (48%) of teachers indicated 
concern about what students were doing while others were being assessed and reported they felt 
their students were not receiving high quality instruction. While teachers reported they like the 
data, they did not feel it was worth the time spent assessing and away from instruction. 

All K-3 teachers who were surveyed or interviewed indicated concern for time spent 
doing benchmark assessment and progress monitoring. More specifically, K-2 teachers who were 
interviewed stated they spent 24 hours on average assessing during benchmark periods, while K- 
2 teachers who were surveyed indicated 26 hours. Third grade teachers’ survey results indicated 
28 hours. Among the teachers surveyed, 74% indicated progress monitoring takes up too much 
instructional time. Teachers were concerned with the frequency required to assess students at 
“yellow” level (every 20 days) and “red” level (every 10 days) because the frequency did not 
allow sufficient time to teach deficit skills to make improvement between monitoring. 
Participants reported they would prefer to spend the time currently devoted to mCLASS: 
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Reading 3D assessments for guided reading and/or small group instruction (42%) and more or 
higher quality instruction (38%). 

Overall, 50% of third grade teachers surveyed were “slightly” or “not at all satisfied” 
with the overall utility of Amplify in providing data to identify and intervene with students 
experiencing reading difficulty. Teachers reported insufficient time to use benchmarking and 
progress monitoring data, 52% and 46%, respectively. While 72% agreed or strongly agree that it 
is a meaningful use of time to analyze benchmarking data, 89% of teachers surveyed thought that 
the Amplify assessment instrument took too much time away from instruction and 64% thought 
the infonnation gained through Amplify assessment was not worth the loss in instructional time. 

Anderson, J., Hutchins, B., Howse, R., Amwake, L., Dufford-Melendez, K., & Wynn, L., et al 
(2015). Read to Achieve Evaluation Year Two: An Evaluation of the Implementation of Read to 
Achieve Assessmen t and Reding Camp. SERVE Center at the University of North Carolina at 
Greensboro. 

The complete (2014) Year One Report can be accessed at: 
http://www.serve.Org/uploads/files/RtA%20_report_SERVE_9.5.14.pdf. 

b. Amplify’s Accuracy on Universal Screening 

You note in your letter that a core component of the Read to Achieve program is 
universal screening. Recent research conducted by REL Southeast (2019) examined longitudinal 
data from mCLASS: Reading 3D to predict outcomes on grade 3 EOG. In the study, data were 
analyzed using classification and regression tree (CART) analysis to identify decision rules for 
identifying students in grades K-3 as at risk or not at risk for reading problems at the end of 
grade 3. The following administration points of the mCLASS 3D reading were used as predictors 
of grade 3 EOG: kindergarten middle of the year (MOY), grade 1 beginning of the year (BOY) 
and MOY, grade 2 BOY, and grade 3 BOY. 

The study used a longitudinal statewide sample of 91,855 students entering kindergarten 
in the 2014-2015 school year and completing grade 3 in 2017-2018. Results revealed a large 
number of students were misidentified as likely to be non-proficient in grade 3 in all of the 
models, with the exception of a model in grade 3. While prediction models beginning with grade 
1 MOY were acceptable for identifying students who are likely to be proficient by the end of 
grade 3, more infonnation was needed in order to accurately identify students who are at risk of 
not being proficient at the end of grade 3. Less than 80 percent of students were accurately 
identified as at risk of poor perfonnance (that is, the sensitivity rate was found to be below a 
recommended minimum standard of .80). Grade 3 BOY with North Carolina’s grade 3 
beginning-of-grade assessment as an additional predictor was the only model that correctly 
classified over 80 percent of students as at risk or not at risk for reading problems at the end of 
grade 3 (that is, both the sensitivity and specificity rates, respectively, were found to be greater 
than .80). Findings of the study are limited to the cohort and assessments included in the analysis 
and may differ with additional cohorts and/or assessments. However, the study findings point to 
the need to supplement the mCLASS student scores with infonnal, curriculum-based 
assessments when setting instructional objectives and differentiating classroom instruction. The 
REL Southeast is in the process of preparing a full report with the study findings. 
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c. Amplify is Only One of Many Opportunities for Teacher-Student Engagement 

You claim in your letter that Amplify’s “direct engagement with students’ thinking 
during the assessment also provides more reliable results.” But teachers should be listening to 
their students reading daily, regardless of the benchmark assessment used three times yearly. 
While different, both Istation and Amplify offer opportunities for teachers to monitor progress, 
detennine areas of need, and work closely with students. 

Additionally, with implementation of Amplify’s mCLASS, teachers have not been 
allowed to assess their own students for MOY or EOY due to fidelity issues. Therefore, the 
student is assessed by another teacher with whom the student may not have rapport, which can 
certainly impact the reliability of results. On the other hand, with Istation, students can be 
assessed by their own teachers in their own classrooms. 

d. Decreasing Reading Scores in North Carolina 

In your letter, you also state that “mCLASS’s benefits have not only been seen in North 
Carolina schools, but nationwide. Amplify has partnered with schools across the country and has 
a demonstrated history of successful implementations.” While we do not dispute that Amplify’s 
mCLASS has some strengths, reading scores have gone down in North Carolina since Amplify 
was fully implemented across the state. (See Exhibit H). 

e. Amplify is Open to Inappropriate Actions by Teachers 

Despite years of training on the assessment, school districts continue to report issues with 
inappropriate actions (such as sharing secure test materials) by teachers using mCLASS, 
resulting in testing irregularities with mCLASS. In 2018-19 alone, 23 districts reached out to the 
Department for guidance in handling these issues. While formal investigation results are still 
coming in, at least 14 testing irregularities have been formally reported to the Department’s 
accountability division for 2018-19, with retesting being the usual remedy. 

f. Amplify Technical Issues Present Challenges for North Carolina Educators 

Educators across the state, tech directors at school districts, the Department’s IT division, 
and K-3 Literacy personnel experienced significant challenges in 2018 due to issues with 
Amplify incorrectly assigning students to class rosters. Amplify failed to timely address the 
situation, making matters worse. There were 250 mCLASS related issues created in our DPI-RtA 
Jira ticketing system and October of 2018 during BOY (Note: In September and October, the 
issues were imported into Jira from the classing Department spreadsheet in Sharepoint, so a 
resolution is not reported, but data are included). Because of the magnitude of mCLASS related 
issues in October, the Department had to hire a temporary technical specialist to manage and 
remedy the volume of Amplify issues. 
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Below is a breakdown of the time Amplify took to resolve issues reported each month 
from September 2018 to July 2019, with tables for both normal priority and high/critical priority 
issues. 


On average, it took 21 days for Amplify to resolve normal priority issues, ranging from 
0-46 days for resolution. However, a requirement from the existing contract with Amplify 
(BAFO 3 Contract) for resolving a nonnal priority issue states the following: “The Contractor 
will provide a preliminary status update and, to the extent practical, preliminary remedy to the 
State not later than the next business day after the problem is initially logged with technical 
support.” (See Exhibit I, pg. 20). 


mClass Issues Resolution Report — RTA-DPI Project in Jira for Read to Achieve - DPI Classing Issues 

Below is a chart of the number of mClass Issues Created, Issues Resolved, Total Resolution Time, and 
Average Resolution Time. 

Period Created Issues Resolved Total Resolution Time Avg. Resolution Time 


September 2018 

105 

0 

0 

0 

October 2018 

39 

1 

21 

21 

November 2018 

12 

49 

2230 

45 

December 2018 

7 

4 

186 

46 

January 2019 

20 

15 

201 

13 

February 2019 

5 

6 

102 

17 

March 2019 

3 

4 

48 

12 

April 2019 

3 

2 

24 

12 

May 2019 

18 

13 

189 

14 

June 2019 

46 

50 

473 

9 

July 2019 

5 

6 

154 

25 


(Note: all times represent number days before resolution) 


Furthermore, the average resolution time for Amplify for high priority/critical issues was 
9 days, ranging from 0-43 days for resolution. Please note the requirement from the contract with 
Amplify in effect in 2018-19 (BAFO 3 Contract) for resolving a high to critical priority issue 
states the following: “The Contractor will provide a preliminary update and, to the extent 
practical, a preliminary remedy to the State within one (1) hour after the High Priority problem is 
initially logged with technical support.” (See Exhibit I, pg. 20). 


35 


mClass Issues High to Critical Resolution Report - RTA-DPI Project in Jira for Read to Achieve - DPI 
Classing Issues 

Below is a chart of the number of High to Critical mClass Issues Created, Issues Resolved, Total 
Resolution Time, and Average Resolution Time. 


Period 

Created 

Issues Resolved 

Total Resolution Time 

Avg. Resolution Time 

September 2018 

0 

0 

0 

0 

October 2018 

3 

0 

0 

0 

November 2018 

5 

5 

75 

15 

December 2018 

2 

2 

87 

43 

January 2019 

12 

11 

155 

14 

February 2019 

2 

6 

102 

17 

March 2019 

3 

3 

20 

6 

April 2019 

0 

0 

0 

0 

May 2019 

7 

4 

31 

7 

June 2019 

42 

44 

138 

3 

July 2019 

5 4 5 1 

(Note: all times represent number days before resolution) 


Amplify’s past performance with the State and untimely provision of services leave much 
to be desired. Amplify’s unreliable reading diagnostic tool threatens to delay the identification of 
poorly performing students, which ultimately delays providing these students with the additional 
help they need. 

g. No Viable Reading Camp Solution 

An important component of the Read to Achieve legislation is the opportunity for 
students to attend summer reading camps to improve their skills. Amplify does not provide a 
viable solution for these reading camps because it only has an offering for BOY, MOY, and 
EOY periods in a given school year. Amplify’s common practice since the state of North 
Carolina began using mCLASS has been to roll the entire state into the next school year and then 
invalidate and delete all TRC data following the conclusion of a reading camp and prior to the 
start of BOY. 

Yes, delete data. This process was challenged this year when it was discovered and 
Amplify justified deleting data because they had always done it this way. It became a major issue 
for the 2018-2019 school year and beyond due to new requirements to report 1st and 2nd grade 
data, which was not required in previous years. In previous years, 3rd grade data were reported, 
then invalidated and deleted once PowerSchool rostering data file was reactivated at the 
beginning of the year. This was not possible for 1st and 2nd grade, however, because they would 
continue to use mCLASS in subsequent years. When pressed to provide an alternative solution 
for reading camps that provides an “isolated environment,” it was discovered that Amplify 
“shares their environment” with all clients making it difficult for Amplify to create an isolated 
instance to meet the needs for collecting and reporting data for reading camps. It should be noted 
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that Istation offered a solution within 48 hours of the request that will also flag the 3/4 transition 
students for continued monitoring and never delete data. 

h. Lexile Level Issues 

Amplify identifies texts of their Atlas Kits with Lexile levels. Based on the levels 
provided by Amplify, TRC level Q was identified as the text level for third grade students to 
demonstrate grade level proficiency. Based on the Lexile level of 770, level Q most closely 
aligned with NC End of Grade Lexile of 725. Department and local district staff have raised 
concerns about inaccuracies in the levels of the books provided by Amplify. This “mis-leveling” 
means that students’ reading proficiency was tested using books that were too hard, or too easy, 
for them. As a result, teachers and administrators are getting inaccurate information on student 
reading abilities and making instructional and promotional decisions for students based on that 
information. Examples of concerns include: 

• Nearly half of Amplify test books do not have a Lexile Level identified on the common 
search engine Lexile.com. MetaMetrics (Lexile.com headquarters) is well known among 
educators as identifying Lexile Levels of texts. 

• A book that Amplify identified as Level Q (Lexile 770) is listed as Level M (Lexile 570) 
on Lexile.com, potentially allowing students to pass third grade without reaching a high 
enough Lexile level to read at grade level. 

• A book that Amplify identified as Level C was actually Level E. This may have affected 
kindergartners who are supposed to reach at least a Level C to be considered proficient. 

On Lexile.com, the Lexiles are very different within the same band of letters in the Atlas 
TRC Inventory. Many of the texts, within the Atlas Kits, cannot be found using a common 
search engine, and the majority of the identifiable books are mis-leveled. As of April 16, 2019, 
32 of the 74 books from levels A-Z in the Amplify Atlas Kits had no measured listing at 
Lexile.com. 
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